Let y = (x² + ax + b)/(x² + 2x + 3)

=> y(x² + 2x + 3) = (x² + ax + b)

=> y(x² + 2x + 3) - (x² + ax + b) = 0

=> yx² + 2xy + 3y - x² - ax - b = 0

=> x² (y - 1) + (2y - a)x + (3y - b) = 0

=> -{x² (1 - y) + (a - 2y)x + (b - 3y)} = 0

=> x² (1 - y) + (a - 2y)x + (b - 3y) = 0   .................1

Now, differentiate w.r.t. x, we get

(-x² - 2y - 3)*(dy/dx) + 2x(1 - y) + a - 2y = 0

Since the range of the y is specified one of them is maximum and other minimum set

So, dy/dx = 0

=> 2x(1 - y) + a - 2y = 0

=> x = (a - 2y)/{2(y - 1)}

Put value of x in equation of 1, we get

     [(a - 2y)/{2(y - 1)}]² * (1 - y) + (a - 2y)* [(a - 2y)/{2(y - 1)}] + (b - 3y) = 0

After simplification, we get

=> (a - 2y)² = 4(y - 1)(3y - b)   ..........2

This implies at y = -5 and y = 4

Put y = -5 in equation 2, we get

=> (a + 10)² = 4(5 - 1)(-5 * 3 - b) 

=> (a + 10)² = 16(-15 - b)

=> (a + 10)² = -240 - 16b  .............3

Put y = 4 in equation 2, we get

=> (a - 8)² = 4(4 - 1)(3 * 4 - b) 

=> (a - 8)² = 12(12 - b)

=> (a - 8)² = 144 - 12b  ..............4

Multiply 3 in equation 3 and 4 in equation 4 and subtract, we get

=> 3(a + 10)² - 4(a - 8)² = -720 - 48b - 576 + 48b

=> 3(a² + 20a + 100) - 4(a² - 16a + 64) = -1296

=> 3a² + 60a + 300 - 4a² + 64a - 256 = -1296

=> -a² + 124a + 44 = -1296

=> a² - 124a - 44 -1296 = 0

=> a² - 124a - 1340 = 0

=> a² - 134a + 10a - 1340 = 0

=> a(a - 134) + 10(a - 134) = 0

=>(a - 134)*(a + 10) = 0

=> a = 134, -10

Put a = -10 in equation 4, we get

      (-10 - 8)² = 144 - 12b

=> (-18)² = 144 - 12b

=> 324 = 144 - 12b

=> -12b = 324 - 144

=> -12b = 180

=> b = -180/12

=> b = -15

Similarly, we can find another value of b

Now, a² + b² = (-10)² + (-15)²     {a = -10, b = -15}

=> a² + b² = 100 + 225

=> a² + b² = 325

Similarly, we can find another value of a² + b²